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Abstract This presentation illustrates the framework of 
processing performance of the faculty of the University of 
Camerino. The evaluation criteria are explained and the 
technological structure that allows automatic performance 
assessment available online anywhere and anytime. The 
designed framework is usually applied to the performance 
evaluation of staff working for Universita di Camerino 
(ITALY). It has provided excellent results which were 
positively judged by those who underwent the assessment, 
who also found that it portrayed their work quite properly. 
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1. Introduction 

Evaluation of academic quality, productivity and 
performance has a key role on academic improvement of any 
academic unit such as an academician, group of 
academicians, department, institution or country. 
Performance and productivity evaluation is always a hot 
issue for all kinds of organizations regardless of their 
business or operational area. Moreover, quality management 
systems have been developed and been applied in order to 
control, evaluate, assure and improve quality hence 
performance and productivity. As in other industries and 
businesses, similar management systems started to take place 
in the academic world. 

The importance of academic quality evaluation is strongly 
related with goals and objectives of an academic unit. On the 
other hand, the evaluation helps being aware of current status 
in terms of productivity and performance. Thus, it can be 
inferred that the evaluation leads to being aware of current 
status and it supports setting meaningful short term and 
long-term goals. In other words, how meaningful goals set 
depends on how successful an evaluation conducted. 
Successful evaluation also depends on how successful 
indicators selected. In short, indicators, evaluation, 
awareness and goals are heavily correlated terms. [7] 

An alternative approach to the provision of research and 
teaching would be a combination of institutional 


diversification together with individual specialization. In this 
scenario, although the university aims at providing both 
high-quality teaching and high-quality research, some 
faculty members specialize in one of them. In any case, 
institutional or individual specialization is rather exceptional 
and most universities worldwide seem to adopt the synergic 
approach, based on the complementarity between teaching 
and research, motivating their faculty members to perform a 
mix of the aforementioned tasks together with certain 
amount of administrative duties. Taking this observation as a 
positive indication on the perceived synergic relationship 
among different types of academic duties, the normative 
question is whether the actual mix of the corresponding 
inputs and outputs is the optimal, or, alternatively, whether it 
could be improved by a redefinition of the obligations and 
objectives set by governments and university authorities [1] 
Most academic positions in universities require three 
types of activities, namely teaching courses, conducting 
research in their respective fields and performing 
administrative duties. The UAF framework has been 
implemented to evaluate how academics perform their 
activities. However, data quality (i.e., that is correctness, 
completeness and uniqueness) can also benefit from the 
adoption of the UAF framework. [2] 

UNICAM has decided to implement a system to monitor 
and evaluate the activities of individual teachers/researchers 
in compliance with the European Charter for Researchers 
and the recommendations by the European Commission 
published on the Official Journal of the European 
Commission of 11th March, 2005 \ which specifies what 
follows: ‘Employers and/or funders should introduce for all 
researchers, including senior researchers, 
evaluation/appraisal systems for assessing their professional 
performance on a regular basis and in a transparent manner 
by an independent (and, in the case of senior researchers, 
preferably international) committee. Such evaluation and 
appraisal procedures should take due account of their overall 
research creativity and research results, e.g. publications, 
patents, management of research, teaching/lecturing, 
supervision, mentoring, national or international 


1 http://eur4ex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32005H025 
1 
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collaboration, administrative duties, public awareness 
activities and mobility, and should be taken into 
consideration in the context of career progression. The 
evaluation of these activities requires collection of data. The 
university assigns to different entities (such as administrative 
staff and secretaries) different data to be maintained. This is 
done in order to enhance data quality that is correctness, 
completeness and uniqueness. Although uniqueness may be 
guaranteed (each data has single managing entity) 
completeness and correctness are not guaranteed. For 
instance, the entity can make mistakes while entering the 
data; the entity can miss some update of data (if needed). The 
UAF framework not only evaluates academics’ activities but 
also improves data quality. More precisely, each entity (that 
is being monitored) is highly interested in checking his own 
data for inaccuracies that can negatively affect his 
performance index. In this regard the lesson we learned 
while using the UAF framework is that releasing data and 
assessment criteria to the monitored entities will greatly 
improve data quality. 

When implementing UAF the information’s completeness 
should be functional to the definition of any subsequent 
evaluations aimed at achieving different and specific 
objectives. Different weighting should be assigned to 
researchers’ activities and the monitoring of individual 
researchers should also be integrated by other kinds of 
information, such as details of the facilities used, for 
example. The method also provides indications for the 
assessment of individual researchers aimed at measuring 
their overall commitment. The assessment considers all the 
activities mentioned in the monitoring, which are assigned 
different weighting. 

Monitoring is carried out for each calendar year taking 
into account the previous three years (except for those cases 
where appointment/employment started during the 
three-year period in question). The evaluation allows for a 
comparison of the cost against the ’value’ produced by the 
researcher in terms of commitment and results obtained. At 
the end of the assessment, each researcher will be given a 
rating, which defines their commitment and that may be 
translated into months. Such a rating will then be multiplied 
by the value of the unit of measure (€ 5,833) 2 and compared 
with the average value for the reference category the 
researcher belongs to 3 . 

2. Researcher Rating Card 

2.1. Monitoring and Measurement of Research Activity 


2 The researcher’s commitment is assigned a specific value using 
Euro/month as a unit of measure (€/M) and is calculated by dividing 70,000 
(average cost in Euro of an Associate Professor) by 12 (€ 5,833). 

3 The cost of one unit of staff has been quantified by the Italian Ministry for 
University and Research (MIUR) as follows: Annual cost of a Full Professor, 
Euro 100,000.00; Annual cost of an Associate Professor, Euro 70,000.00; 
Annual cost of a University Researcher/Temporary Researcher, Euro 

50,000.00. 


This section includes both the research products and the 
relevant and verifiable activities related to the research. 
Currently, there are about 30 different items that are taken 
into consideration and that are going to be evaluated. 

According to the type of item, some reference values come 
into play and, possibly, some corrective parameters too. 

As regards scientific articles published in journals, within 
bibliometric sectors, a reference value is based on an 
estimate of the time devoted to their production by a 
researcher. It was estimated that a researcher may devote 
3/10 of his/her working time to such an activity. 

After obtaining (source ANVUR 4 ) the number of articles 
published over a 10-year period by a researcher working 
within a given Discipline Scientific Sector (such a number 
being defined as ‘m’), taking into account that 3 out of the 10 
years in question were devoted to scientific production and 
that 1 point is equivalent to 1 month of work, each article was 
given a 36/m score. Such a score implicitly includes the 
evaluation of author numerosity too. 

A corrective factor for such a score is the positioning of 
the journal in question within the quartiles of the citation 
ranking (source Journal Citation Report - Web of Science). 

Within non-biometric sectors, such as, for example, law, 
architecture, economics and social sciences, the reference 
value is similar to the one of biometric sectors. The only 
difference is the estimate of the time devoted by the 
researcher to scientific production. 

As a corrective factor, the journal’s ranking also affects 
other items, such as reviews, judgment/sentence remarks, as 
well as the role or function assigned, i.e. whether the person 
is an editor or a member of an editorial board. 

As regards other types of publications, such as conference 
proceedings, encyclopedia items, patents, etc., the author 
numerosity factor comes into play too. 

Peculiar items pertaining to some disciplines are also 
considered, such as the production of geological maps, 
projects, architectural compositions, and the organization of 
scientific events, expositions and exhibitions. 

In this section, even the researcher’s commitment 
concerning the presentation of research projects at various 
levels (i.e. European, national or regional ones) is rewarded 
too, even if no funding is linked to such projects. 

In this last case, the source of data is the database of the 
central administration office that is devoted to the 
accountancy and administrative management of research 
projects. 

2.2. Monitoring and Assessment of Educational Activities 

The following activities are monitored and evaluated: 

• The production of study books used for teaching 
university students; 


4 Agenzia Nazionale di Valutazione del Sistema Universitario e della 
Ricerca. http://www.anvur.org 



728 


Unicam Activity Framework (UAF) 


• The production of written texts or chapters for 
university study books; 

• Thesis supervision; 

• The supervision of PhD students; 

• The hours devoted to institutional or post-graduate 
lecturing on site, at any associated premises, or within 
other courses held abroad and managed by the 
university. 

As regards the hours of lecturing, the score for every hour 
is 0.025 points 5 (see note) and 5 corrective factors have also 
been envisaged, which are respectively linked to: 

1. The evaluation by students expressed in questionnaires 
concerning the teacher’s performance in terms of 
subject knowledge, presentation quality, availability, 
punctuality; 

2. Lecturing activity on site or off site; 

3. Number of students in the classroom; 

4. Language used for lecturing; 

5. Lecturing within e-learning courses. 

The score given to thesis supervision is linked to the role 
played by the teacher as supervisor, advisor or tutor, and to 
the type of study course the student enrolled for, i.e. bachelor, 
master, 5-year degree. 

2.3. Monitoring and Assessment of Funding 

This refers to the funds managed by UNICAM. The 
relevant figures are to be equally split and distributed 
amongst the entire internal researcher involved in the funded 
research. 

The evaluation concerns the quota of EU funding that the 
university obtains as overheads, under a ‘PQ’ code, other 
international funding, national funding obtained through 
competitive programs (e.g. PRIN and FIRB), research 
funding from government or private bodies. Different scores 
and overheads are assigned according to the type of funding. 

Data are automatically captured from the accountancy and 
administration management system used for research 
projects. 

2.4. Monitoring and Assessment of Institutional 
Functions 

This concerns the work carried out by teachers on a 
voluntary basis that, most of the times, is not paid for. As in 
all previous cases, a score is assigned to every activity 
according to the time devoted to it (calculated in months 
within a given year). 


The roles and functions considered include the following 
ones: rector, prorector, functions delegated by the rector, 
management of or accountability for the organization of 
scientific or teaching facilities. Besides the institutional 
functions, other roles are also taken into consideration, such 
as being a member of high-profile, national or international 
institutions or commissions. 

The teacher should record his/her functions in the 
framework. For every kind of function there is a person 
responsible of the relevant data that is in charge of 
confirming (through a special interface within the framework) 
what the teacher in question stated about his/her functions. 

Some non-cumulative mechanisms have also been set up. 

3. Framework Development 

The framework helps to develop comprehensive Business 
Intelligence solutions. 

The platform is process-centric because it uses Workflow 
engine as central controller. Workflow engine uses process 
definitions to define the business intelligence process that it 
executes on the platform. The process may be easily 
customized and new processes can also be added. The 
platform includes components and reports that may be used 
to analyze the performance of such processes. 

The platform is solution-oriented because its operations 
are specified in the definition process and in the operation 
documents that provide details of every single activity. 

These processes and operations together define the 
solution for the business intelligence problem. The solution 
may be easily added to business procedures outside the 
platform. The solution definition may include a number of 
processes and operations. 

The platform is composed of: 

• Business Intelligence Framework - which provides 
logging, auditing, security, scheduling and ETL 
procedures, as well as web services, attribute 
repository and rules engine; 

• Business Intelligence Components - including 
components for reporting, analysis, workflow, 
dashboards and data mining; 

• Business Intelligence Workbench - which is a set of 
design and administration tools enabling analysts 
and developers to create reports, dashboards, 
analysis models and business intelligence processes; 

• Desktop Inboxes - which deliver notices generated 
by processes and reports. 

The following image shows the system’s architecture. 


5 The value of 0.025 comes from the assumption that 40 hours of front 
lecturing are equivalent to one month of full-time work (1 point), including 
lecture preparation time, office hours, and mid-year and final tests and 
exams. 
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RESEARCHER RATING CARD 

DASHBOARDS 

REPORTS 


3.1. Business Intelligence Framework 

The framework includes some special functions related to 
the flow of activities. The latter are governed by a workflow 
engine and are stored in the EVALUATION database that is 
used for data analysis, reporting purposes, and for the 
dashboards available to end users (mainly the university’s 
management and teacher evaluators). 

The main functions concern mining data from several 
sources that are shown in the relevant picture as 4 databases 
named: ADMIN, DID A, HR, R&S. 

ADMIN is the database used to obtain information related 
to the projects managed by teachers that received funding, 
and to the proposals for projects without funding. 

As already mentioned in Sections 3.1 and 3.3, both items 
are evaluated. The procedure for data mining also includes a 
processing phase for those projects whose funding amount is 
not known in advance. In such cases the amount is calculated 
for accountancy purposes by selecting and summing up the 
single payments received during the period considered. 

DIDA represents the source of those data used to carry out 
the evaluations concerning the lecturing activity (Section 
3.2). In this case, we are dealing with the most relevant 
amount of data, as they include the information concerning 
teachers’ and students’ careers, as well as the questionnaires 
completed by the latter. 

In this case, the data mining process is highly selective and 
is focused on filtering information in order to extract only 
what is needed within the time interval in question. To do so, 
the system also uses some differential updating mechanisms. 

R&S contains the information related to the teachers’ 


scientific publications that is used to extract the data needed 
to evaluate the research aspects specified in Section 3.1. This 
database will be fed by each teacher, every time that he/she 
uploads his/her publications through the IRIS * 6 application, 
which has recently replaced the old U-GOV 7 . 

The IRIS database is quite important as it was used 
nationwide by ANVUR 3 too for its VQR project, in order to 
evaluate the results of scientific research carried out by 
universities and research organizations. It is also going to be 
used to allocate (according to an award program) the 
ordinary funds that support the institutional work of the 
above players. 

HR contains the information concerning the institutional 
work of every teacher, as well as other activities which are 
going to be considered for evaluation purposes that could not 
be obtained from the above-mentioned databases. 

This database was created in parallel with the framework 
in question, together with the application that enables users 
to record their institutional work and that, at the same time, 
can also be accessed by those in charge of confirming the 
teachers’ statements. The procedure includes a number of 
reports that are meant to support activity verification and 
checking. 

Other information comes from heterogeneous sources. For 
example, the journal ranking (source Journal Citation Report 
- Web of Science) is available as an electronic data sheet. 


6IRIS - Institutional Research Information System. 

http://www.cineca.it/it/content/IRIS 

7 U-GOV is an integrated IT system for the governance of universities and 
research organizations, http://www.cineca.it/it/content/il-sistema-u-gov 
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The main process for the implementation of the workflow 
currently includes 40 activities. Each activity includes 
several individual operations. Following the necessary 
authentication, the process begins with the verification of the 
databases involved in the reading or writing activity. Once 
that phase has been successfully completed, the other 
operations are activated in a parallel or sequential mode. For 
any activities that require the connection to a database, the 
authentication phase is repeated. If an error occurs, the 
process stops and, anyway, the administrators are informed 
about the results of the whole process, as well as about every 
single activity carried out within the main process. 

The job is scheduled to be carried out based on a given 
frequency that may vary according to the requirements 
envisaged, and that may also be increased in conjunction 
with evaluation processes. 


To implement the workflow within the framework, Kettle 
software was used. Pentaho Data Integration (PDI, also 
called Kettle) is the component of Pentaho responsible for 
the Extract, Transform and Load (ETL) processes. Though 
ETL tools are most frequently used in data warehouses 
environments, PDI can also be used for other purposes and, 
in this particular instance, by using KETTLE’S terminology 
representation the main workflow process was implemented 
through a job that includes the 40 or so changes that, in turn, 
are composed of an articulated series of processes. 

The workflow system was installed on a LINUX server 
and Cron is used for the scheduling process, together with 
the operating system’s shell. The picture below shows the 
first part of one of the main jobs. In the following picture the 
details of a change made within a job are shown. 
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3.2. Business Intelligence Components 

As regards the, Kettle functions are used to regulate the route throughout the various processes. A workflow can be 
graphically built by dragging the single items in the job and by linking them to each other as needed. The following picture 
shows a simplified part of the process concerning the uploading of journal ranking data on the EVALUATION database (see 
Section 3.1). The original data are provided every year by CRUI 8 as electronic data sheets. Before being imported into the 
EVALUATION database, data need to be picked from the sheet and undergo some special processing. This activity is carried 
out through a process named Transformation 2. But before the process is activated some necessary conditions need to be 
verified through the process named Transformation 1. During the latter process, a given number of parameters are checked 
and, subsequently, the presence of the electronic data sheet within the file system is also verified (Process File Exist 1). 


► 


5T4RT 



The available reports have been developed through the Community Edition of Pentaho Reporting that includes a suite of 
Open Source tools needed to create reports in PDF, Excel, HTML, Text, Rich-Text-File, XML and CSV formats. The 
reporting functions are carried out thanks to the JfreeReport 9 engine that is owned and sponsored by Pentaho. 

The main framework end user is the university management that prefers to have the results in a web page format, and as 
reports or dashboards. 

The picture below shows an example of a web page containing the Research Rating Card as described in the previous 
sections. 


8 CRUI is an association composed of Italian state and private universities that entered into an agreement with the Thomson Reuters publishing company to 
access the databases of Web of Science and Journal of Citation Reports (JCR). 

9 JFreeReport is a free Java report library. It has the full on-screen print preview data obtained via Swing's TableModel interface based on report definitions, 
output to the screen, printer or various export formats, support for servlets. 
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Within the EVALUATION database some data structures were developed according to the ROLAP model. Some tools that 
may be used to carry out multidimensional analyses are also available. 

In particular, an interface developed through the SAIKU tool enables users to explore data sources by applying the familiar 
‘drag and drop’ technique through a web page. 

Finally, the university management can also use a series of dashboards that provide a concise view of the data, which are 
aggregated at various levels: by individual teacher, by role (researcher, senior fellow, full professor), by school, seen as a 
result of the aggregation, by individual facility, by faculty and by department. 

The dashboard of a teacher is shown in the picture below as an example. 

Users may select a given academic year and a particular teacher, and the relevant results will be displayed on the dashboard 
as: 

1. A bar graph containing the following information: 

a. The scores obtained by the teacher in the 4 sections displayed as a bar graph (green bars). 

The scores can be compared with the lines showing the teachers’ average score for each section. 

b. The scores of the scientific area of the teacher (blue line) 

c. The scores of the same role (red line) 

d. The scores of the same Discipline Scientific Sector (yellow line) 

e. The scores of the whole university (black line) 
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Dati detente 


34 


33- 
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2. A Gauge Chart that compares the teacher’s overall score (shown by the indicator with the minimum, medium and 
maximum overall score) calculated amongst all the teachers evaluated. Red and yellow identify respectively the scores 
that are above and below average. 


Punti totale del docente 



3. Four gauge charts show the scores obtained by the teacher in each one of the 4 sections by comparing them with 
minimum, medium and maximum scores obtained by all the teachers in the same sections. 
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4. A Line Chart that shows the scores obtained in each section by the teacher in the selected year and, for comparison 
purposes, in the three previous years. 



5. A table showing the scores obtained in the selected year. 



Punti 


RiCfrrcs 


34.4713 

DkJattica 


31,535 

Finanziamenti 


1+4337 

Inesnchi 


fi .1777 

A set of bar charts showing: 

The scores obtained by the teacher in each section 

The average scores of teachers working: 
within the same school 
within the same scientific area 
in the same role 

within the same Discipline Scientific Sector 
within the whole university 
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3.3. Business Intelligence Workbench 

The set of tools available with the Pentaho Data 
Integration (PDI, also called Kettle) and the Pentaho BI 
Server Community Edition allowed us to provide the 
developers, and sometimes analysts too, with the tools 
needed to create whatever was required for the framework 
development. 

Pentaho’s BI Server or BA platform allows you to access 
business data in the form of dashboards, reports or OLAP 
cubes via a convenient web interface. Additionally it 
provides an interface to administer your BI setup and 
schedule processes. 

In particular, using the Pentaho BI Server as a reference, 
we used the CDE Community Dashboard Editor and all the 
technology underneath it - namely CDF, CDA and CCC - 
allows the development and deployment of Pentaho 
Advanced Dashboards. Community Dashboard Editor (CDE) 
was born to simplify the CTools Dashboards’ creation, 
edition and rendering processes, and it's a very powerful and 
complete tool, combining front end with data sources and 
custom components in a seamless way. 

Community Data Access (CDA) is the plugin designed for 
accessing data. 

CCC stands for Community Chart Components, the 
CTools charting library. 

As regards the multidimensional analysis function, in 
addition to the Saiku tool used for the analysis rendering, to 
build the OLAP cubes we used Mondrian, which is included 
in the suite that is an open source OLAP (online analytical 
processing) server, written in Java. It supports the MDX 


(multidimensional expressions) query language and the 
XML for Analysis and olap4j interface specifications. It 
reads from SQL and other data sources and aggregates data 
in a memory cache. 

3.4. Desktop Inboxes 

The Inbox Alerter is an executable that needs to be 
installed on the machines of the users that wish to take 
advantage of its functionality. The Inbox Alerter provides 
many ease-of-use features such as: 

• Notification of new workflow tasks 

• Notification of report delivery 

• Management of off-line content 

The Inbox Alerter uses an RSS feed provided by the 
Pentaho Server. Any RSS Reader that supports authenticated 
feeds can be used to receive notifications from the server. 
For off-line content management the Pentaho Inbox Alerter 
is required. The Pentaho Inbox Alerter will be provided 
based on the demand for client platform. 

4. Conclusions and Future Development 

Academic jobs require simultaneous performance of a 
variety of duties, including teaching, research and 
administration captured in a large number of alternative 
performance measures available on each one of them. 
Assuming that teaching is a university’s major contribution 
to the society, we have addressed the question whether 
teaching quality is affected by other academic duties. 
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The system that was implemented allows for the 
evaluation of many aspects of the work carried out by 
university researchers in relation to the many duties that 
globalization has brought about for the development of 
academic organizations. 

Therefore, the relative value of university teachers is not 
determined just by their scientific value, but also by their 
ability to teach, to follow the study route of their students and 
to attract funding for the development of applied research. 
Last but not least, the management skills of each player 
within a university system have become a fundamental 
parameter for the overall evaluation of performance, 
especially because this kind of activity has become 
extremely important for university education, in every 
country. 

Moreover, the literature we reviewed [9, 10, 4, 8] clearly 
demonstrates that: 

• In general, the quality of university teaching is 
positively affected by published research. Specifically, 
research output improves teaching quality for small 
amounts of research, below the median performance in 
our sample, reaches a maximum and decays slowly for 
higher values of research output. That is, usual amounts 
of research are positive for teaching, while high 
research output may hinder teaching quality. In fact, 
two thirds of the professors could improve their 
teaching by doing more research. 

• The quantity of teaching has a positive effect on 
teaching quality for relatively low teaching loads. The 
quality peaks around the average teaching load 
observed in our sample. In summary, moderate amounts 
of teaching, are beneficial for the quality, while higher 
teaching loads may decrease teaching quality. Since the 
average teaching load was 8 in the period under study, 
the observed arrangement does not seem to be far from 
the optimum. 

• The quality of teaching is not affected by heavy 
administration duties. This may be because these 
appointments entail mandatory reductions in the 
workload to compensate for administrative work. 

• The quality of teaching is negatively affected by light 
administrative duties. This may arise because many of 
these duties do not entail a compensating reduction in 
teaching loads. 

• The quality of teaching is positively related to the 
participation in say, relevant committees, such as 
self-evaluation commissions, creation/reform of degree 
curricula, etc. 

• The quality of teaching is positively associated to the 
elaboration of books or multimedia teaching materials 
with ISBN. 

• Faculty members at the Education Dept, obtain better 
teaching results than the rest of the sample. 

The developed framework may be added to other projects 
which are suggested in relevant literature [3, 5, 6]. However, 
some of its features make it quite different: 


• The framework includes all the work carried out by a 
university researcher; 

• It is directly connected with the university’s IT system 
(that, in Italy, is supplied by the Ministry of Education 
to 90% of Italian universities); 

• Given its natural evolution, different evaluation 
weighting criteria for various activities may be 
potentially set up, through a Business Intelligence tool; 

• The performance dashboard, which enables the 
management to know how their university is going in 
real time. 

As future work we are planning to quantify the 

enhancement in data quality. 
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